Hitherto, in a system for transmitting a video signal of moving picture to remote place, for example, as in the television conference system, the television telephone system, etc., or a system for recording a video signal of moving picture onto a picture recording medium such as an optical disc, a magnetic disc or a magnetic tape, etc., or reproducing a recorded video signal of moving picture, there is adopted for the purpose of efficiently utilizing transmission path (or picture recording medium), a scheme to implement so called efficient encoding to a video signal by making use of correlation between lines or correlation between frames that video signal has to reduce redundancies in the spatial axis direction and the time axis direction to transmit only significant information, thus to improve transmission efficiency.
For example, in encoding processing in the spatial axis direction (hereinafter intra-frame coding processing), e.g., correlation between lines of a video signal is utilized as shown in FIG. 7A. In the case of attempting to transmit respective pictures PC1, PC2, PC3 . . . constituting a moving picture at times t1, t2, t3 . . . , picture data to be transmission-processed is caused to undergo one-dimensional coding, e.g., within the same scanning line, or a picture is divided into, e.g., a plurality of blocks to allow picture data of respective blocks to undergo two-dimensional coding to thereby carry out data compression, thus to improve transmission efficiency.
Moreover, in coding processing in the time axis direction (hereinafter referred to as inter-frame coding processing), inter-frame correlation of video signal is utilized to determine, by so called predictive coding, for example, picture data PC12, PC23 . . . comprised of deficiencies (so called predictive errors) of picture data every corresponding pixels between adjacent pictures PC1 and PC2, PC2 and PC3 . . . in succession to transmit these picture data PC12, PC23, . . . to thereby carry out data compression, thus to improve transmission efficiency.
Thus, as compared to the case where all picture data of pictures PC1, PC2, PC3 . . . are transmitted, a video signal can be transmitted by extremely lesser data quantity.
Further, in the predictive coding in the above-described inter-frame coding processing, motion compensated prediction is used, e.g., in macro block units in order to further improve efficiency. Namely, e.g., in the case where a person at the central portion of picture moves, or the like, motion (movement) of an object moving in the picture is detected to correct position of picture data used for prediction in the former picture by that motion to carry out predictive coding, thereby making it possible to improve coding efficiency. However, even when such motion compensated prediction is employed, many data must be transmitted with respect to the portion where an object moves and appears from behind. In view of this, not only motion compensation in the above-described forward direction, but also motion compensation in backward direction or in both directions of forward and backward directions are carried out in combination, thereby making it possible to further improve coding efficiency.
In actual terms, as shown in FIG. 8A, in macro blocks of frame data F0, F1, F2, F3 of the 0th, first, second, third . . . frames of a video signal of moving picture to be transmitted, in the case where there took place changes of pictures as respectively indicated by motion vectors x0, x1, x2, x3 . . . between frames in succession, device on the transmitter side designates frames at intervals of a predetermined number of frames (e.g., every other frame), i.e., second, fourth . . . frames as interpolation frames to implement so called predetermined interpolation frame processing to these interpolation frames as shown in FIG. 8B to thereby generate transmit interpolated frame data F2X, F4X . . . . Further, with respect to non-interpolation frames, the device on the transmitting side implements a predetermined coding processing to frame data F1, F3 . . . to generate transmit non-interpolated frame data F1X, F3X . . . .
For example, difference SP2 (predictive error) between motion compensated frame data F3 and F2, difference SP3 between motion compensated frame data F1 and F2, and difference between frame data obtained by implementing interpolation processing to motion compensated frame data F1, F3 and frame data F2 are respectively determined in macro block units to compare difference (data) SP1 of frame data F2 and those differences. Then, data having minimum data quantity generated of those data SP1˜SP4 is caused to be transmit interpolated data F2X in macro block units. Similarly, transmit interpolated data F4X . . . with respect to respective interpolation frames are generated. Further, e.g., DCT processing and variable length coding processing, etc. are implemented to frame data F1, F3 . . . of non-interpolation frames to generate transmit non-interpolated frame data F1X, F3X . . . .
The transmit non-interpolated frame data F1X, F3X . . . and transmit interpolated frame data F2X, F4X . . . are transmitted to the device on the receiving side as transmit data along with motion vectors x0, x1, x3 . . . .
On the other hand, the device on the receiving side implements decoding processing corresponding to coding processing on the transmitting side to transmit data (transmit non-interpolated frame data F1X, F3X . . . , transmit interpolated frame data F2X, F4X . . . , data of motion vectors x0, x1, x3 . . . ), thus to reproduce frame data F0, F1, F2, F3 . . . . As a result, motion compensation is implemented not only in forward direction but also in backward direction or in forward and backward directions, thereby making it possible to further improve coding efficiency.
Picture encoding apparatus and picture decoding apparatus having the above-described function will now be described.
This picture encoding apparatus comprises, as shown in FIG. 9, a pre-processing circuit 61 for separating an input video signal VD into luminance signal and color difference signal, analog/digital (hereinafter referred to as A/D) converting circuits 62a, 62b for respectively converting the luminance signal and the color difference signal from the pre-processing circuit 61 into digital signals, a frame memory group 63 for storing luminance data and color difference data (hereinafter referred to as picture data) from the A/D converting circuits 62a, 62b, a format converting circuit 64 for reading out picture data from the frame memory group 63 in accordance with block format, and an encoder 65 for implementing efficient coding to picture data of block from the format converting circuit 64.
In operation, pro-processing circuit 61 separates input video signal VD into luminance signal and color difference signal. A/D converting circuits 62a, 62b respectively converts luminance signal and color difference signal into luminance data and color difference data each comprised of 8 bits. Frame memory group 63 stores these luminance and color difference data.
Format converting circuit 64 reads out, in accordance with block format, picture data (luminance data, color difference data) stored in the frame memory group 63. Encoder 65 encodes the picture data thus read out by a predetermined efficient coding to output bit stream.
This bit stream is delivered to picture decoding apparatus 80 through transmission media 70 comprised of transmission path or picture recording media such as, optical disc, magnetic disc or magnetic tape, etc.
This picture decoding apparatus 80 comprises, as shown in the FIG. 9 mentioned above, decoder 81 corresponding to the encoder 65, format converting circuit 82 for converting picture data reproduced by the decoder 81 into frame format, frame memory groups 83 for storing picture data from the format converting circuit 82, D/A converting circuits 84a, 84b for converting luminance data, color difference data which have been read out from the frame memory group 83 into analog signals, and post-processing circuit 85 for mixing luminance signal, color difference signal from the D/A converting circuits 84a, 84b, thus to generate output video signal.
Decoder 81 decodes bit stream by decoding corresponding to efficient coding of encoder 65 to reproduce picture data of block format. Format converting circuit 82 converts this picture data into frame format to store it into frame memory group 83.
D/A converting circuits 84a, 84b respectively convert luminance data and color difference data which have been read out from frame memory group 83 into luminance signal and color difference signal. Post-processing circuit 81 mixes these luminance signal and color difference signal, thus to generate output video signal.
In actual terms, pre-processing circuit 61 and A/D converting circuits 62a, 62b convert luminance signal and color difference signal into digital signal as described above to reduce quantity of data so that the numbers of pixels become equal to one half of those of luminance signal in upper and lower directions and in left and right directions with respect to the luminance signal thereafter to implement time axis multiplexing processing thereto to deliver luminance data and color difference data thus obtained to frame memory group 63.
From frame memory group 63, luminance data and color difference data are read out in accordance with block format as described above. Namely, e.g., picture data of one frame is divided into N slices as shown in FIG. 10A. Each slice is caused to include M macro blocks as shown in FIG. 10B. Each macro block is composed of luminance data Y1, Y2, Y3, Y4 of four luminance blocks consisting of 8×8 pixels adjacent in upper and lower directions and in left and right directions and color difference data Cb, Cr of color blocks consisting of 8×8 pixels in a range corresponding to these four luminance blocks. From frame memory group 63, luminance data and color difference data are read out so that picture data are successive in macro block units within slice and are successive in order of Y1, Y2, Y3, Y4, Cb, Cr within macro block. Picture data which have been read out in accordance with block format in this way are delivered to encoder 65.
Encoder 65 comprises motion vector detecting circuit 101 as shown in FIG. 11. This motion vector detecting circuit 101 detects, in macro block units, motion vector of picture data delivered thereto in accordance with block format. Namely, motion vector detecting circuit 101 detects, in macro block units, motion vector of current reference picture by forward original picture and/or backward original picture stored in frame memory group 83. Here, detection of motion vector is carried out such that minimum one of absolute value sums of differences between frames in macro block units is caused to be corresponding motion vector. The motion vector thus detected is delivered to motion compensating circuit 113, etc., and intra-frame differences in macro block units are delivered to intra-frame/forward/backward bidirectionally predictive judging circuit 103.
This intra-frame/forward/backward/bidirectionally predictive judging circuit 103 determines predictive mode of reference block on the basis of this value to control predictive coding circuit 104 so as to carry out switching of intra-frame/forward/backward/bidirectional prediction in macro block units. Predictive coding circuit 104 comprises adding circuits 104a, 104b, 104c and selecting (changeover) switch 104d, and is operative so that when predictive coding mode is intra-frame coding mode, it selects input picture itself, and when predictive coding mode is forward/backward/bidirectionally predictive mode, it selects differences (hereinafter referred to as difference data) every pixels of input picture data with respect to respective predictive pictures, thus to deliver the selected data to DCT circuit 105.
DCT circuit 105 implements DCT processing to input picture data or difference data in block units by making use of the two-dimensional correlation of video signal to deliver coefficient data thus obtained to quantizing circuit 106.
The quantizing circuit 108 quantizes coefficient data by using quantization step size (quantization scale) determined every macro block or slice to deliver quantized data thus obtained to variable length coding (hereinafter referred to as VLC) circuit 107 and inverse quantizing circuit 10B. Meanwhile, quantization step size used for this quantization is determined so as to take a value such that transmitting buffer memory 109 which will be described later does not break by providing feedback of buffer residual of transmitting buffer 109. This quantization step size is also delivered to VLC circuit 107 and inverse quantizing circuit 10B.
VLC circuit 107 implements variable length coding to quantized data along with quantization step size, predictive mode and motion vector to deliver them to transmitting buffer memory 109 as transmit data.
The transmitting buffer memory 109 temporarily stores transmit data thereafter to read out it at a predetermined bit rate to thereby smooth transmit data to output it as bit stream, and to feed quantization control signal in macro block units back to quantizing circuit 108 in accordance with residual data quantity remaining in the memory to control quantization step size. Thus, transmitting buffer memory 109 adjusts data quantity generated as bit stream to maintain data of appropriate residual (remaining capacity) (data quantity such that no overflow or underflow takes place) within the memory. For example, when data residual of transmitting buffer memory 109 increase to allowed upper limit, transmitting buffer memory 109 allows quantization step size of quantizing circuit 108 to be large by quantization control signal, thus to reduce data quantity of quantized data. On the other hand, when data residual of transmit buffer memory 109 decrease down to allowed lower limit, transmitting buffer memory 109 allows quantization step size of quantizing circuit 106 to be small by quantization control signal to thereby increase data quantity.
In this way, bit stream outputted from buffer memory 109 is delivered to picture decoded unit 80 through transmission media 70 comprised of a transmission path or a picture recording medium such as optical disc, magnetic disc, or magnetic tape etc. at a predetermined bit rate as described above.
On the other hand, inverse quantizing circuit 108 inverse-quantizes quantized data delivered from quantizing circuit 106 to reproduce coefficient data (quantization distortion is added) corresponding to output of the about-described DCT circuit 105 to deliver the coefficient data to Inverse Discrete Cosine Transform (hereinafter referred to as IDCT) circuit 110.
The IDCT circuit 110 implements IDCT processing to the coefficient data to reproduce picture data corresponding to input picture data in the intra-frame coding mode, and to reproduce difference data corresponding to output of predictive coding circuit 104 in the forward/backward/bidirectionally predictive modes, thus to deliver it to adding circuit 111.
When predictive coding mode is the forward/backward/bidirectionally predictive modes, the adding circuit 111 is supplied with motion-compensated predictive picture data from motion compensating circuit 113 which will be described later to add the motion-compensated predictive picture data and difference data to thereby reproduce picture data corresponding to input picture data.
The picture data reproduced in this way is stored into frame memory 112. Namely, inverse quantizing circuit 108˜adding circuit 111 constitute a local decoding circuit to locally decode quantized data outputted from quantizing circuit 106 to write decoded picture thus obtained into frame memory 112 as forward predictive picture or backward predictive picture. The frame memory 112 is comprised of a plurality of frame memories. Bank switching of the frame memory is carried out. In correspondence with picture to be encoded, single frame is outputted as forward predictive picture data, or is outputted as backward predictive picture data. Moreover, in the case of bidirectional prediction, forward predictive picture data and backward predictive picture data are, e.g., averaged. The averaged data thus obtained is outputted. These predictive picture data are entirely the same pictures as pictures reproduced by decoder 81 which will be described later. Picture to be processed next is caused to undergo forward/backward/bidirectional predictive coding on the basis of this predictive picture.
Namely, picture data which has been read out from frame memory 112 is delivered to motion compensating circuit 113. This motion compensating circuit 113 implements motion compensation to predictive picture data on the basis of motion vector to deliver the motion-compensated predictive picture data to predictive encoding circuit 104 and adding circuit 111.
Decoder 81 will now be described.
To decoder 81, bit stream is inputted through transmission media 70 is inputted. This bit stream is inputted to Variable Length Decoding (Inverse Variable Length Coding) (hereinafter referred to as IVLC) through receiving buffer 201. The IVLC circuit 202 reproduces quantized data, motion vector, predictive mode and quantization step size, etc. from bit stream. These quantized data and quantization step size are delivered to inverse quantizing circuit 203. Motion vector is delivered to motion compensating circuit 207, and predictive mode is delivered to adding circuit 205.
The operation of inverse quantizing circuit 203˜adding circuit 205 is the same as that of local decoding circuit of encoder 61, the operations of frame memory group 206, motion compensating circuit 207 are respectively the same as those of frame memory 112 and motion compensating circuit 113 of encoder 61. On the basis of quantized data, motion vector, predictive mode, quantization step size, decoding is carried out. As a result, reproduction picture data is outputted from adding circuit 205.
As described above, in the conventional apparatus, coding bit rate of bit stream generated at encoder 65 is caused to be fixed in correspondence with transfer rate of transmission media 70. Under this limitation, quantity of data generated, i.e., quantization step size of quantizing circuit 106 in encoder 65 was controlled. In other words, for example, a control was conducted such that when pictures of complicated pattern are successive, quantization step size is caused to be larger to suppress quantity of data generated, while when simple patterns are successive, quantization step size is caused to be smaller to increase quantity of data generated so that buffer memory 109 does not produce overflow or underflow, thus to maintain a fixed rate.
Accordingly, in the conventional apparatus, when complicated pictures are successive, quantization step size is caused to be larger, so picture quality is deteriorated, while when simple pictures are successive, quantization step size is caused to be smaller. As a result, uniform picture quality cannot be obtained through the entirety.
In addition, in the case of recording bit stream onto a picture recording medium of a limited data capacity, in order to avoid extreme deterioration of picture quality with respect to pictures of complicated pattern, a fixed rate of high rate such that picture quality of such complicated picture is not injured must be applied to the entirety, resulting in decreased recording time.